5 

SUPREM-DSMC:  A  NEW  SCALABLE,  PARALLEL,  REACTING,  MULTIDIMENSIONAL  DIRECT 

SIMULATION  MONTE  CARLO  FLOW  CODE 


David  Campbell,  Dean  Wadsworth,  Douglas  VanGilder 
ERC,  Incorporated 
Air  Force  Research  Laboratory 
Edwards  AFB,  CA 

Ingrid  Wysong 

Air  Force  Research  Laboratory 
Edwards  AFB,  CA 

Carolyn  Kaplan,  David  Mott 
Naval  Research  Laboratory 
Washington,  DC 


ABSTRACT 

An  AFRL/NRL  team  has  recently  been  selected  to  develop  a  scalable,  parallel,  reacting, 
multidimensional  (SUPREM)  Direct  Simulation  Monte  Carlo  (DSMC)  code  for  the  DoD  user  community 
under  the  High  Performance  Computing  Modernization  Office  (HPCMO)  Common  High  Performance 
Computing  Software  Support  Initiative  (CHSSI).  This  paper  will  introduce  the  JANNAF  Exhaust  Plume 
community  to  this  three-year  development  effort  and  present  the  overall  goals,  schedule,  and  current 
status  of  this  new  code. 


INTRODUCTION 

Since  being  introduced  by  Bird1  over  30  years  ago,  the  basic  DSMC  technique  has  matured  and 
evolved,  and  a  number  of  codes  exist  today  that  are  used  on  a  regular  basis  in  a  variety  of  applications. 
DSMC  codes  are  applied  most  often  in  the  gas  dynamic  flow  regime  in  which  the  Knudsen  Number  (Kn) 
is  in  the  range  0.01<Kn<10,  where  Kn  is  the  ratio  of  the  gas  mean  free  path  to  the  characteristic  length  of 
the  problem.  This  is  a  regime  where  the  continuum  assumptions  inherent  in  the  Navier-Stokes  equations 
are  invalid.  The  DSMC  technique  is  a  well  established  and  reliable  method  for  simulating  such  gas  flows. 
These  flows  cover  a  very  broad  range  of  space  and  time  scales,  arising  in  low  density  gases  that  occur  in 
high  altitude  applications  and  in  high  density  gases  in  very  small  devices  such  as  microelectromechanical 
systems  (MEMS).  However,  there  is  no  standard,  fully-functional  parallel  DSMC  code  presently  available 
to  the  JANNAF  user  community.  The  need  for  a  modern,  portable,  scalable,  documented,  user-friendly 
and  readily  available  DSMC  code  has  been  recognized  by  the  DoD  HPCMO,  which  has  funded  a  three- 
year  development  effort  to  build  such  a  code.  An  AFRL/NRL  team  has  been  assembled  to  accomplish 
this  major  task.  Combining  the  resources  and  complementary  expertise  of  these  two  research  groups, 
this  project  began  on  January  1 ,  2000. 

The  goal  of  this  effort  is  to  develop  and  transition  to  the  user  community  a  modern,  scalable  DSMC 
code  based  on  the  integration  of  state-of-the-art  collision  models  with  advanced  parallelization  methods, 
gridding  algorithms  and  data  structures.  While  the  paramount  characteristics  of  the  code  will  be 
robustness  and  ease  of  use,  other  goals  include  the  following  code  capabilities: 

•  Parallel,  scalable  solution  of  CPU-intensive  3-D,  unsteady  flows 

•  Accurate  representation  of  and  resolution  of  highly  nonequilibrium  chemical  and  collisional  processes 

•  Documented  databases  of  key  reaction  rates  and  molecular  constants 
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•  Automated  grid  adaptation  and  related  capabilities  to  allow  use  by  a  broad  range  of  non-expert  users 

•  Standardized  and  documented  code  operation  and  software-design  methodology 

.  Easily  extendable  user  interface  and  data  structures  to  allow  enduring  use  and  continued  code 

enhancement  and  customization 

These  code  capabilities  should  provide  a  valuable  tool  for  analysis  of  a  wide  range  of  exhaust  plume 
problems,  as  well  as  find  use  in  microelectromechanical  device  development,  hypersonic  flight  and 
reentry  vehicle  analysis,  and  investigation  of  spacecraft  environments. 

GENERAL  DISCUSSION  OF  SUPREM-DSMC  DEVELOPMENT 

DEVELOPMENT  PHILOSOPHY 

Although  both  groups  participating  in  this  project  have  ongoing  relevant  research  efforts,  the 
SUPREM-DSMC  project  is  not  focused  on  advancing  the  state  of  the  art,  but  is  primarily  concerned  with 
providing  a  product  in  a  reasonable  time  frame  that  is  usable  by  a  non-expert  for  real-world  problems. 
This  mandates  that  the  code  be  robust,  have  internal  checking  capabilities,  and  provide  the  user 
feedback  on  the  solution  quality.  Attention  will  be  focused  on  lowering  risk  by  implementation  of  proven 
computation  models  and  techniques,  with  capabilities  for  adding  new  models  and  techniques  as  they  are 
developed. 

Many  applications  of  interest  introduce  conflicting  demands  on  the  software  suite,  and  potentially 
involve  physical  processes  and  features  for  which  existing,  validated,  or  robust  DSMC  models  are  not 
available.  Several  decision  points  involving  tradeoffs  and  compromises  are  explicitly  scheduled  early  in 
the  program  to  determine  what  specific  models  and  techniques  to  use  in  the  various  modules  of  the  code. 
An  additional  challenge  involves  the  need  for  a  relatively  large  degree  of  automation  (expert  systems)  in 
the  code  to  allow  use  by  novice  users.  Features  such  as  local  grid  and  time  step  adaptation,  with 
appropriate  heuristic  control  methods,  will  be  an  integral  part  of  the  software.  It  is  also  recognized  that 
there  will  be  a  significant  challenge  in  obtaining  and  maintaining  portability  and  robustness  with  the 
desired  level  of  novice-user  convenience,  as  no  such  software  has  been  demonstrated  for  the  demanding 
applications  of  interest. 

One  vital  additional  criterion  during  the  development  of  this  code  is  that  continued  participation  and 
advice  of  DSMC  experts  and  the  user  community  be  sought  and  encouraged.  An  advisory  group  of 
experts  in  the  field  has  already  been  assembled  to  serve  as  technical  advisors  and  alpha  and  beta 
testers.  The  advisory  board  will  review  the  technical  approach  during  the  initial  phases  of  the  project,  and 
will  periodically  participate  in  status  review  and  problem  resolution.  Technology  transfer  to  the  user 
community  is  an  integral  component  of  the  project  and  will  be  accomplished  through  a  series  of  meetings, 
training  workshops  and  frequent  dialogue  between  the  software  developers  and  applications  experts.  It  is 
strongly  desired  that  any  interested  users  contact  one  of  the  team  members  as  soon  as  possible. 

A  web  site  has  been  set  up  for  dissemination  of  information  to  the  interested  technical  community: 
http://www.lcp.  nrl.navv.mil/~ckaplan/chssi.html. 

QUESTIONAIRE  RESULTS 

One  of  the  first  actions  of  the  SUPREM-DSMC  team  was  to  compose  a  questionnaire  that  was  sent 
to  DSMC  practitioners  and  potential  SUPREM-DSMC  users  to  obtain  information  on  the  types  of 
applications  of  interest  and  the  required  code  capabilities.  Most  respondents  were  experienced  DSMC 
users  or  developers.  They  were  interested  in  a  wide  range  of  applications,  with  a  majority  primarily 
concerned  with  rocket  exhaust  plumes  and  reentry  vehicle  flow.  Due  to  the  preponderance  of  interest  in 
plumes,  the  respondents  were  most  interested  in  using  DSMC  simulations  to  obtain  detailed  flowfield 
information  at  steady  state  in  complex  2-dimensional  and  3-dimensional  geometries  with  complex 
chemistry  at  elevated  gas  temperatures  and  with  inert  solid  surfaces.  They  also  felt  that  the  features  of 
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Table  1:  Capabilities  Timeline 


Software  Release 

a 

P 

IOC 

Release  Date 

June  2001 

June  2002 

Dec  2002 

Geometry 

2D 

3D 

Grid 

Fixed 

Adapting 

Gas  Species 

Monatomic,  Multispecies 

Arbitrary  #  of  Modes, 

Continuous/Quantized, 

Databases 

Collision 

Elastic 

Boundary 

Conditions 

Source 

Constant 

Freestream 

Spatial  Varying 

Time  Dep. 

Wall 

Fully  Accom 
(Maxwell) 

Incomplete  Accommodation, 
Spatial  Varying,  Outgassing, 
Time  Dependent  Motion 

User  Interface 

Test  File 

GUI 

Automation 

Time  Step 

Global,  Input 

Local  Collision  Frequency, 
Memory  Limits,  Collision 

Spatial  Weights 

Global,  Input 

Separation,  Local  Cell 

Population 

Host  Architecture 

SP2,  02K,  SP3 

SP2,  02K,  SP3,  ES-40  (Alpha) 

Documentation 

a  Doc 

P  Doc 

IOC  Doc 

Demos 

2D  Missile  Body/Plume 

2D  MicroChannel 

3D  Missile  Body/Plume 

3D  MicroChannel 

Hierarchical  and  modular  input  files  and  code  structures  will  be  used  to  allow  for  user-customization 
through  standardized  application  programming  interfaces  (APIs)  to  the  general  code  data  structures. 
Solution  quality  enhancement  will  be  made  possible  by  inclusion  of  automated  or  user-controlled 
adaptation  features  in  the  code.  These  include  features  relating  to  numerical  solution  quality  (such  as 
grid  cell  size,  timestep  size,  or  statistical  scatter)  and  physical  realism  (such  as  the  complexity  of  the  gas 
representation  or  the  collision  models  used). 

Initial  interfaces  to  the  codes  will  be  through  keyword-based  text  files.  Later  versions  will  utilize 
optional  GUIs.  The  pre-processor  will  perform  input  file  error  checking,  job  setup,  and  initialize  the 
SUPREM-DSMC  data  structures.  The  main  code  will  be  designed  for  batch  operation  on  large  HPC 
computers,  but  the  provision  will  made  for  interactive  operation.  The  post-processor  will  allow  for 
extraction  and  reformatting  of  relatively  arbitrary  user-defined  data  for  plotting  and  analysis. 
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The  input  files  to  the  codes  will  consist  of  geometry,  boundary  condition,  gas  species  and  collision 
definition,  and  execution  control.  These  files  will  be  designed  to  provide  both  definition  of  the  problem 
itself,  and  a  concise  documentation  of  the  processes  actually  used  in  the  code.  In  this  way,  the  user  can 
directly  understand  (or  at  least  must  explicitly  define)  what  documented  features,  techniques,  or  models 
are  being  used,  without  recourse  to  the  "blackbox"  source  code.  The  files  will  be  hierarchical,  such  that  a 
basic  case  can  be  easily  set  up,  but  a  more  experienced  user  can  define  increasingly  complex  models  to 
be  used,  or  phenomena  to  be  resolved.  The  execution  control  file  will  describe  the  basic  features  of  the 
job  such  as  desired  spatial  and  temporal  accuracy,  memory  or  run-time  constraints. 

GRIPPING 


SUPREM-PSMC  will  use  a  two-level,  3-P  cartesian  computational  grid.  The  Level  1  grid  may  be  a 
uniform  or  clustered  grid,  and  will  be  initialized  at  the  beginning  of  the  computation.  As  the  calculation 
proceeds,  a  Level  2  adapted  grid  will  be  generated,  where  the  adaptation  criteria  may  be  the  mean  free 
path  or  number  density,  or  possibly  a  local  flow  gradient  or  other  detailed  property.  The  grid  will  be 
defined  and  stored  independently  of  the  input  geometry.  The  geometry  will  be  defined  in  terms  of  simple 
geometric  primitives  for  simple  bodies,  or  surface  triangulations  for  more  complex  bodies.  Boundary 
surfaces  will  be  discretized  on  the  Level  2  grid  to  facilitate  the  molecule  movement  process,  whereas  load 
balancing  will  be  done  at  Level  1 . 

The  Alpha-version  of  the  code  will  model  2-B  stationary  geometries,  where  the  user  specifies  the 
characteristics  of  the  Level  2  grid  and  surface  elements.  The  data  structures  will  be  sized  and  allocated 
based  on  a  user-specified  grid  dimension  or  size,  and  load  balancing  will  be  static,  based  on  the  grid 
volume.  Spatial  weight  factors  will  be  constant  based  on  an  estimated  value,  but  may  be  globally 
adjusted  to  satisfy  memory  (number  of  simulation  particles)  constraints. 

The  Beta-version  of  the  code  will  include  3-P  stationary  geometry  and  2-P  rigid  body  motion.  The 
Level  2  grid  and  surface  elements  will  be  automatically  generated.  In  addition,  the  Beta-version  will 
include  an  interface  with  a  CAP  or  grid-generation  program  to  generate  the  surface  grid;  in  this  case,  if 
further  grid  refinement  is  necessary,  SUPREM-PSMC  will  generate  the  appropriate  3-P  surface  grid  by 
subdividing  the  input  geometry  surface  triangulation  as  part  of  the  grid  adaptation  process.  Pata 
structures  will  be  sized  and  allocated  based  on  reference  flow  conditions  or  from  an  existing  solution. 
Load  balance  may  be  either  static,  based  on  estimated  work  or  on  an  existing  solution,  or  dynamic,  based 
on  CPU  time.  Computational  cell  structure  will  be  adapted  to  a  local  bulk  flow  property,  such  as  mean 
free  path,  or  adapted  to  a  local  flow  gradient  or  detailed  property,  while  surface  elements  will  be  adapted 
to  a  near  surface  local  flow  property.  Capabilities  for  spatial  weight  factors  will  include:  a  global 
adjustment  to  satisfy  collision  separation  constraints,  a  local  constant  flux  value  (adjusted  with  timestep), 
a  global  adjustment  to  accelerate  transient  computation,  or  a  local  adjustment  to  obtain  desired  local 
molecule  cell  population. 

BOUNPARY  CONPITIONS 


The  boundary  conditions  are  grouped  into  general  types  and  sub-types  to  allow  for  a  wide  variety  of 
models  and  processes  to  be  simulated  for  the  creation  of,  or  reflection  of,  molecules  at  computational 
boundaries.  For  example,  a  wall  reflection  process  can  vary  from  a  simple  diffuse  interaction  in  the  Alpha 
version  of  the  code  to  a  chemical  reaction  process  dependent  on  specific  energy  modes  of  the  colliding 
species  in  later  versions.  Similarly,  capabilities  for  the  general  description  of  inflow,  outflow,  and 
symmetry  boundaries  will  be  included.  Future  enhancements  (possibly  beyond  the  CHSSI  project 
timeframe)  include,  for  example,  capabilities  for  ^P  rigid  body  motion,  and  2-P  and  3-P  generalized  | 
boundary  motion.  © 
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MOVEMENT  OF  MOLECULES 

Molecules  will  be  moved  on  the  Level  2  grid.  Molecules  will  be  moved  in  sub-iterations,  where  the 
displacement  is  limited,  if  necessary,  to  the  boundary  of  the  Level  2  cell  in  which  it  initially  resides.  This  is 
necessary  to  avoid  translation  errors  when  a  molecule  moves  across  Level  2  cells  with  varying  timesteps 
and  to  account  for  spatial  weighting  factors.  Ray  tracing  techniques  will  be  used  to  compute  the  potential 
intersection  of  the  molecule  trajectory  with  boundary  elements.  Communication  buffering  techniques  will 
be  used  to  address  the  movement  of  a  molecule  to  a  computational  cell  which  is  assigned  to  another 
processor. 

The  alpha-version  will  allow  for  the  movement  of  molecules  through  a  2-D  fixed  grid  and  geometry. 
Beta-version  capabilities  include  movement  of  molecules  through  a  2-D  adapted  grid  and  through  a  3-D 
fixed  and  adapted  grid.  In  addition,  the  beta-version  will  allow  for  movement  of  molecules  through  a  grid 
with  variable  spatial  weighting  factors  and  timesteps,  and  through  a  2-D  grid  in  which  there  is  a  defined 
geometrical  boundary  motion. 

GAS  SPECIES.  COLLISIONS  AND  CHEMICAL  REACTIONS 

The  gas  species  representation,  the  collision  processes  they  can  experience,  and  the  models  used  to 
describe  the  interaction  will  be  defined  in  the  gas  species  input  file,  rather  than  being  hardwired  internal  to 
the  code.  The  species  definition  can  include  an  arbitrary  number  of  continuous  or  quantized  internal 
energy  modes,  and  thus  allows  for  representation  of  atomic  through  arbitrary  polyatomic  species.  Each 
quantized  internal  energy  mode  may  include  a  specification  of  one  or  more  radiative  lifetimes,  and  the 
code  will  allow  for  the  effects  of  unimolecular  (radiative)  depletion  of  excited  modes.  If  desired,  specific 
quantized  levels  can  be  defined  as  separate  “species”  to  improve  representation  via  species  weighting 
factors,  or  to  allow  rudimentary  representation  of  state-specific  processes.  SUPREM-DSMC  will  not 
include  coupled  radiation,  and  it  is  not  planned  to  have  the  code  output  radiance  maps,  but  every  effort 
will  be  made  to  make  the  output  compatible  with  existing  post-processor  radiance  codes.  It  is  anticipated 
that  these  features  will  be  added  in  follow-up  work  after  the  initial  CHSSI  project  in  completed. 

The  collision  events  are  described  by  chemical  reaction  equations  which  define  the  species  involved, 
the  energy  modes  that  contribute  to,  or  take  part  in,  the  collision,  and  the  models  used  to  carry  out  the 
process.  An  arbitrary  number  of  potential  collision  processes  can  be  defined  for  a  given  species  pair. 
The  event  selection  is  done  using  the  null  collision  scheme,  where  the  event  probability  is  related  to  the 
relative  magnitude  of  the  event  (i.e.,  approximately  the  event  cross  section),  and  only  one  event  occurs 
per  collision  (but  it  can  involve  relatively  arbitrary  changes  in  the  colliding  species). 

The  collision  events  are  nominally  categorized  as  elastic,  inelastic,  or  general  reaction  processes. 
Several  basic  DSMC  models  will  be  pre-defined  for  each  category,  but  the  basic  approach  allows  for  the 
consistent  inclusion  of  a  wide  variety  of  processes  and  models  to  be  defined  and  implemented  by  the 
user  as  they  become  available.  For  elastic  collisions,  the  variable  hard  sphere1  or  variable  soft  sphere7 
models  will  be  available.  For  inelastic  processes  involving  the  redistribution  of  one  internal  mode  of  one 
molecule,  a  collision  number  (constant  probability)  scheme  will  be  provided,  where  an  accepted  event 
undergoes  redistribution  of  the  mode  via  the  Borgnakke-Larsen  model.  The  general  class  of  reaction 
events  allows  for  definition  of  arbitrary  changes  to  the  species  and  the  modes  involved  in  the  collision.  A 
generalized  form  of  the  Bird  total  collision  energy  model1  based  on  known  (input)  Arrhenius  rate 
coefficient  data  will  be  provided.  In  this  form,  the  user  may  specify  which  internal  energy  modes  of  each 
reactant  contribute  to  the  reaction  probability,  a  feature  not  included  in  some  earlier  versions  of  this 
model.  Finally,  for  each  of  the  general  event  categories,  allowance  is  made  for  input  of  tabulated  cross 
section  data  for  cases  where  no  analytical  model  of  an  event  is  available  or  suitable. 

The  present  level  of  maturity  of  models  of  chemical  and  collision  events  in  DSMC  precludes  definitive 
determination  of  a  unique  database  which  satisfies  both  computational  efficiency  and  physical  accuracy 
requirements  for  the  general  problem.  Continuing  research,  both  internal  to  and  external  to  the 
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AFRL/NRL  team,  is  expected  to  provide  improvements  in  this  area  and  any  relevant  results  from  these 
efforts  will  be  utilized  to  the  fullest  extent  possible  during  SUPREM-DSMC  development.  To  assist  the 
novice  user,  several  standardized,  documented  gas  databases  will  be  provided  with  the  code.  The 
databases  are  expected  to  combine  various  degrees  of  complexity  of  the  gas  species  and  collision 
representation  to  allow  for  immediate  application  to  specific  problems  involving,  e.g.,  room  temperature 
air,  or  strongly  shocked  atmospheric  species. 

PRESENT  CODE  STATUS 

At  present,  code  development  efforts  have  concentrated  on  assembly  of  templates  for  the  various 
modules  of  the  software.  The  emphasis  is  on  establishing  consistency,  generality,  and  extendability  in 
the  user-interface  (e.g.,  input  file  definition)  and  for  source  code  development  (data  structure 
representation  and  code  modularity).  The  basic  structure  of  the  software  is  expected  to  be  finalized  by 
the  time  of  the  software  acceptance  test  (SAT)  scheduled  for  Q3  FYOO. 

SUMMARY  AND  CONCLUSIONS 

A  new  development  effort  has  begun  that  will  provide  the  user  community  with  a  standard,  scalable, 
parallel,  reacting,  multidimensional  Direct  Simulation  Monte  Carlo  computational  code.  It  is  anticipated 
that  the  rocket  exhaust  plume  community  will  be  a  major  user  group  for  SUPREM-DSMC.  Potential  users 
are  encouraged  to  contact  the  developers  to  establish  a  working  relationship  during  these  early  stages  of 
development. 

Though  a  large  number  of  capabilities  are  planned  for  SUPREM-DSMC,  the  initial  versions  of  the 
code  will  be  relatively  basic.  The  emphasis  in  the  early  phase  of  the  project  is  to  design  the  code  and  the 
user  interface  to  present  a  consistent  and  modular  new  analysis  capability  for  the  greatest  number  of  DoD 
users.  The  improved  capabilities  and  features  will  be  added  incrementally,  eliminating  the  need  for 
complete  rewrite  of  the  codes,  while  ensuring  that  the  time  the  new  user  invests  in  the  initial  versions  will 
contribute  to  accelerated  integration  of  the  final  code  into  his  or  her  research,  design,  testing,  and 
engineering  toolkit. 
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